Apparatus, system and method for using a universal controlling device for displaying a graphical user element in a display device

ABSTRACT

An apparatus, system and method is described for controlling one or more consumer electronic devices that is performed by a smart device. The smart device causes one or more graphical elements, each for performing a particular operation to be displayed. The smart device then causes an action to be performed when the user selects one of the graphical elements, for example by pressing one of the keys on the universal controlling device or by speaking a command.

RELATED APPLICATION INFORMATION

This application claims the benefit of and is a continuation of U.S.application Ser. No. 16/157,907, filed on Oct. 11, 2018, whichapplication claims the benefit of and is a continuation of U.S.application Ser. No. 15/986,002, filed on May 22, 2018, whichapplication claims the benefit of and is a continuation of U.S.application Ser. No. 15/784,637, filed on Oct. 16, 2017, the disclosuresof which are incorporated herein by reference in their entirety.

BACKGROUND

Universal controlling devices, such as for example, remote control typedevices which are configured to issue commands to a multiplicity ofappliances of different type and/or manufacture, and the features andfunctionality provided by such controlling devices are well known in theart.

For commanding the operation of multiple appliances using a singleuniversal controlling device, a conventional controlling devicetypically comprises multiple device mode states. In each device modestate, the universal controlling device is configured to command theoperation of one or more designated appliances. For example, a typicaluniversal remote control may be placed into one of the multiple devicemode states through actuation of a corresponding device mode key. Inthis example, the user relies on the key labels to determine the properdevice mode. The actuation of such a device mode key functions toconfigure the universal remote control to transmit command codes to theone or more appliances that have been designated to the device modestate corresponding to the actuated device mode key.

One drawback of such an arrangement is that is incumbent upon the userof the controlling device to be aware of, and if necessary reselect, anappropriate device mode state prior to issuing operating commands toeach of the controlled appliances. All too frequently users may forgetor overlook this step, resulting in an unintended action being performedby the wrong device.

A further shortcoming of such controlling devices is that users mustalways be looking at the input elements, for example keys on a universalremote control, may initiate different command functions in differentdevice modes. The prevalence of such input key clutter is yet anothersource of user confusion. Therefore, it would be desirable to provide auniversal controlling device that allows a user to navigate and controlappliance functions without the input key clutter, mode confusion, andinput key labels, thereby eliminating the need to look down at eachinput key label to perform the desired operation.

SUMMARY

Described hereinafter are examples of an apparatus, a system, and amethod for controlling one or more consumer electronic devices with auniversal controlling device. In one embodiment, a universal controllingdevice has functional components that comprise an electronic storagemedium having processor-readable code embodied therein for storing aplurality of unique key ids where each unique key id is associated witha particular input key of the universal controlling device, input keysfor receiving user input, a communication interface for sending uniquekey ids to a smart device and to a processor coupled to the electronicstorage medium, the input keys, and the communication interface, forexecuting the processor-readable code.

In operation, the processor-readable code causes the processor todetermine that an input key was touched by a user. In response to theprocessor determining that a first input key was touched, the processorsends, via the communication interface, a first unique key id associatedwith the touch of the input key that causes a consumer electronic devicetemplate associated with a first consumer electronic device to bedisplayed by the smart device, where the consumer electronic devicetemplate has the functional components that comprise a set of graphicaluser interface elements, where each graphical user interface elementassociated with a particular action of the consumer electronic deviceand where the processor determines that a second input key was pressed.In response to determining that the second input key was pressed, theprocessor sends via the communication interface, a second unique key idto the smart device, the second unique key id that causes a selection ofa first graphical user interface element from the set of graphical userinterface elements that causes the smart device to have the consumerelectronic device perform the action associated with the first graphicaluser interface element.

In one embodiment, a smart device is described, coupled to a universalcontrolling device, for controlling one or more consumer electronicdevices with a universal controlling device, comprising, an electronicstorage medium having processor-readable code embodied therein forstoring a plurality of consumer electronic device templates, where eachtemplate comprises a graphical representation of a set of actionsavailable to a user to control a particular consumer electronic device,each graphical user interface element associated with one of the actionsand further for storing operational commands each associated with one ofthe actions, the operational commands in conformity with a protocol usedby a particular consumer electronic device associated with the template,a communication interface for receiving an input from the universalcontrolling device, and a processor that is coupled to the electronicstorage medium and the communication interface for executing theprocessor-readable code.

In operation, the processor-readable code causes the processor toreceive, via the communication interface, a first key id from theuniversal controlling device representative of a input key of theuniversal controlling device that was touched by the user, determine afirst consumer electronic device template associated with the first keyid from the plurality of consumer electronic device templates stored inthe electronic storage medium, the first consumer electronic devicetemplate comprising a plurality of first graphical user interfaceelements associated with a particular consumer electronic device, causethe first consumer electronic device template to be displayed by adisplay device, receive, via the communication interface, a second keyid from the universal controlling device representative of an input keyof the universal controlling device that was pressed by the user,determine a first operational command associated with the second key id,and performing an action associated with the first operational command.

A method is described for controlling one or more consumer electronicdevices that is performed by a smart device in cooperation with auniversal controlling device, where a processor of the smart devicereceives, via a communication interface, a first key id from theuniversal controlling device. The smart device processor then determinesa first consumer electronic device template from a plurality of consumerelectronic templates stored in a memory where the first consumerelectronic device template comprises a set of graphical user interfaceelements where each graphical user interface element is associated withan action of a particular consumer electronic device. The set ofgraphical user interface elements are then displayed on a display deviceand the smart device processor then receives, via the communicationinterface, a second key id from the universal controlling device wherethe second key id representative of a second key that was pressed by theuser. The smart device processor then determines a first graphical userinterface element from the set of displayed graphical user interfaceelements from the second key id and causes the consumer electronicdevice to perform an action associated with the first graphical userinterface element.

In one embodiment, system for controlling one or more consumerelectronic devices has functional components that comprise, a universalcontrolling device for receiving user input and for determining, that aninput key was touched by a user, a smart device comprising an electronicstorage medium having processor-readable code embodied therein forexecuting the processor-readable code.

In operation, the processor-readable code causes the processor toreceive, via a communication interface, a first unique key id from theuniversal controlling device; determine, a first consumer electronicdevice template from a plurality of consumer electronic templates storedin a memory, the first consumer electronic device template comprising aset of graphical elements where each graphical element is associatedwith an action of a particular consumer electronic device; display theset of graphical elements on a display device; receive, via thecommunication interface, a second unique key id from the universalcontrolling device where the second unique key id representative of asecond key that was pressed by the user; determine a first graphicalelement from the set of displayed graphical elements from the secondunique key id and cause the consumer electronic device to perform anaction associated with the first graphical element.

In one embodiment, a smart device is described that receives from eachof a plurality of media source appliances connected to a respective oneof a plurality of input ports of the first appliance data indicative ofat least one available media content. The smart device uses the dataindicative of the available at least one media content received fromeach of the plurality of media source appliances to cause a plurality ofdisplay elements to be displayed in a display associated with the firstappliance. When the smart device receives a data that is representativeof a one of the plurality of display elements, the smart device uses thereceived data that is representative of the one of the plurality ofdisplay elements and information indicative of the plurality of displayelements caused to be displayed in the display associated with the firstappliance to determine a command to be transmitted from the firstappliance to a target appliance. The smart device then transmits to thetarget appliance, using a communication protocol recognizable by thetarget appliance, the determined command for the purpose of controllinga corresponding operation of the target appliance.

A better understanding of the objects, advantages, features, properties,and relationships of the invention will be obtained from the followingdetailed description and accompanying drawings which set forthillustrative embodiments and which are indicative of the various ways inwhich the principles of the invention may be employed.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various aspects of the describedsystems and methods, reference may be had to the illustrative examplesshown in the attached drawings in which:

FIG. 1 is a block diagram illustrating one embodiment of a system forcontrolling one or more consumer electronic devices.

FIG. 2A is an illustrative example of using a universal controllingdevice for controlling a set top box while watching TV.

FIG. 2B is an illustrative example of how a graphical element ishighlighted when the user touches (but not presses) a centered key.

FIG. 2C is an illustrative example of how the set top box alters theappearance of play/pause graphical element when the user presses thecentered key.

FIG. 3A is an illustrative example of a consumer electronic devicetemplate, displayed by a smart device when a user is watching a movie.

FIG. 3B is an illustrative example of a consumer electronic devicetemplate for the activity of watching the TV.

FIG. 3C is an illustrative example of another consumer electronic devicetemplate.

FIG. 4 is an example functional block diagram of the universalcontrolling device shown in FIG. 1, for controlling one or more consumerelectronic devices.

FIG. 5 is an example functional block diagram the smart device as shownin FIG. 2, for controlling one or more consumer electronic devices.

FIG. 6 is an example flow diagram for a method for controlling one ormore consumer electronic devices, performed by a smart device incooperation with a universal controlling device.

DETAILED DESCRIPTION

Examples described hereinafter relate generally to embodiments of anapparatus, system and method for using a universal controlling devicefor controlling one or more consumer electronic devices.

FIG. 1 illustrates an example system in accordance with the disclosureherein and includes a universal controlling device 102, operated tocontrol the functional operations of one or more consumer electronicdevices such as a COMCAST® brand, set-top-box 108, (hereafter referredto as “STB”) and an APPLE® TV brand streamer 110 (hereinafter referredto as “streamer”). Some consumer devices are Internet-capable, i.e.,they can send or receive content to and/or from sources located withinrange of a local data network, such as a wireless LAN, or sourceslocated in remote locations via the Internet and may operate to someextent interactively and autonomously, (hereafter referred to as, “smartdevices”). In this example, the universal controlling device 102,comprises input keys where each input key comprises one or morecapacitive touch sensors, such as a capacitive sensing area that candetect and measure conductivity, (hereafter referred to as, a “touchsensor”), and where each input key is programmed with a unique keyidentifier (hereafter referred to as a, “key id”). It will beappreciated that the example touch sensor may include any suitabletouch-like input key, including a proximity sensor, voice regognition,etc., and need not be limited to an input key that detects a physicalcontact between a user and the input key. In one embodiment, theuniversal controlling device 102, comprises a dome switch that islocated beneath and is coupled to the capacitive input key structurethat when compressed, such as when the user presses the dome and thedome is collapsed, provide the user with positive tactile feedback.

Generally, the STB 108 and the streamer 110 are connected to a smart TV106, (hereafter referred to as “TV”), via an HDMI cable 112, may also beconnected to a wireless router 114, may use a wireless signal 116 tocommunicate to a wide area network 118, and/or may operate to send datato a coupled remote server 120, coupled to a database 122.

As will be appreciated from the description that follows, the universalcontrolling device 102 is adapted to transmit one or more differentwireless signals, such as signals 104 and 105, for reception by acorresponding plurality of intended target consumer electronic devices.Such transmission may use communication protocols such as Voice over IP(VoIP), IP, Smart Wave (S-WAVE), Wi-Fi, Bluetooth low energy (BLE),RF4CE, ZigBee, Z-wave, Infra-red, 6LoWPAN, Thread, Wi-Fi-ah, 2G, 3G, 4G,NB-IoT, 5G, NFC, RFID, SigFox, etc., as needed to communicate commandsto the target consumer electronic devices.

The universal controlling device 102 may be operated by a user, using afinger to touch the input key on the universal controlling device 102(as opposed to pressing the input key), which causes the universalcontrolling device 102 to send a first touch key id associated with thetouched input key to a coupled smart device, such as the STB 108. TheSTB 108 receives the first touch key id and, in response, causes anelectronic device template (e.g., a graphical user interfacerepresentation of a set of control and/or activity elements that areavailable for that particular consumer electronic device and which maybe based on the current media content that the user is interactingwith), to be displayed by a display device coupled to the STB 108, suchas the TV 106. Similarly, an input key on the universal controllingdevice 102 may be pressed or otherwise suitably selected, such as forinstance an elongated “touch”, “hover-over”, voice command, etc. In thisexample, the input key on the universal controlling device 102 ispressed via use of a dome structure beneath the capacitive keystructure, for sending a second key id associated with the pressed inputkey to a coupled smart device. In some embodiments, the second key idmay be different than the first key id.

In yet another embodiment, the second key id may comprise a key downevent, a key press event, or any registered key value associated withthe input key that was pressed by the user. For example, when the usertouches an input key and then presses the same input key the universalcontrolling device 102 may send the first key id and the key down eventand/or the registered key value to the smart device.

In one embodiment, the universal controlling device 102 may comprise aprocessor and processor-readable code for programming each input keywith a unique key id when the universal controlling device 102 is firstused. In this embodiment, each unique key id is a unique code that isnot associated with the make, model, or product information associatedwith a consumer electronic device. Generally, infrared remote controlssend digitally-coded pulses of infrared radiation designed by thespecific manufacturer such as SONY®, SAMSUNG®, etc. to control functionssuch as power, volume, tuning, temperature set point, fan speed, orother features and different manufacturers of infrared remote controlsuse different protocols to transmit the infrared commands. In thisembodiment, the unique signal comprising the key id may be in one ormore forms of code such as ASCII, Binary, Oct, Dec, Hex or similar codeswhere each signal may be sent by the universal controlling device 102,using one or more communication protocols and where the unique code isnot designed by a particular consumer electronic device manufacturer.

For example, the unique code may comprise a generic or dummy code whichis independent of the make, model, etc. of the consumer electronicdevice. In some embodiments, at least some of the input keys on theuniversal controlling device 102 are not labeled which eliminates theneed to look down at the universal controlling device 102 to control oneor more consumer electronic devices.

In another embodiment, the universal controlling device 102 may compriseone or more accelerometers and/or one or more multiple-axis gyroscopesthat when activated cause the universal controlling device 102 todetermine a first orientation of the universal controlling device 102,where the first orientation corresponds to a first input key location.The first input key location is determined by the functional arrangementor key-meaning associations of the input keys of the universalcontrolling device 102. In this embodiment, the first orientation dataand the key id of the key that the user touched are sent to the STB 108,for determining an orientation correction that is to be performed by theSTB 108, for realigning each of the input keys with the firstorientation. For example, the user may pick up the universal controllingdevice 102 in any orientation causing the gyroscope to send neworientation data to the processor of the universal controlling device102 that causes the processor to perform an analysis using thepreviously stored orientation data and the new orientation data for thepurpose of determining the change in location of the first input key.The universal controlling device 102 then remaps or otherwise reprogramseach of the input keys to correspond with the orientation data. Suchorientation compensation may be performed as described in U.S. PatentApplication No. 2014/0218290, which is incorporated herein by referencein its entirety.

In one embodiment, the consumer electronic device templates may beprovisioned to the STB 108 during a detection operation or provisioningoperation by the server 120. In this embodiment, the controlling device102 initiates a provisioning operation, i.e. at initial power on, thatcauses the STB 108, to perform a connected consumer electronic devicedetection operation. In some embodiments, the server 120, may receiveconfiguration, (e.g. connected consumer electronic device information),from the STB 108, comprising a Universal Control Engine 100, (hereafterreferred to as “UCE”) via a provisioning process which is described infurther examples below.

In another example, a smart device, in a cooperative operation with theuniversal controlling device 102 and the server 120, is configured in aprovisioning process to enroll each consumer electronic device locatedin the user's home and detected by the smart device, such as a learningoperation, and which associates a consumer electronic device to aconsumer electronic device template.

FIG. 2A is an illustrative example of an embodiment for using theuniversal controlling device 102 for controlling the TV 106 when the TV106 is controlled by the STB 108. In this embodiment, the universalcontrolling device 102 comprises a circular touchpad 208 with a centeredkey 210 disposed in the center of the circular touchpad 208. Thecircular touchpad 208 may be considered to be an input key of theuniversal controlling device 102. The STB 108 provides TV content to theTV 106 typically over the cable 112, such as an HDMI cable. Both the TVcontent and the sound volume of the TV 106 content may be controlled bythe STB 108 via the cable 112. When a user touches either the circulartouchpad 208 or the circular key 210 while watching the TV 106, theuniversal controlling device 102 transmits a first key id, indicatingthat the user has touched either the circular touchpad 208 or thecentered key 210. The STB 108 receives the first key id and, inresponse, selects a first consumer electronic device template 200 from anumber of consumer electronic device templates stored in memory (orretrieved from a remote source, such as the remote server 120) andcauses the first consumer electronic device template 200 to be displayedby the TV 106. In this example, the first consumer electronic devicetemplate 200 comprises a set of graphical elements 202, 204, 206, 212,and 214. In some embodiments, the set of graphical elements 202 mayresemble the input keys of a traditional universal remote control,comprising graphical elements representing navigation commands such asfast forward 202, back/return 204, fast reverse 206, information 212,and play/pause 214 as needed for the desired operation.

While illustrated as a circular shape, it is to be understood that otherconsumer electronic device templates may include, but need not belimited to, various geometric shapes which may align to a particularshape of input key layout (e.g., triangular, squared, etc.) and inputkey location of the universal controlling device 102.

As the user moves his or her finger around the circular touchpad 208,the universal controlling device 102 transmits key ids corresponding todifferent areas of the circular touchpad 208, and the STB 108 causescorresponding graphical elements displayed on the TV 106 to becomehighlighted, shaded, or otherwise enhanced. For example, in FIG. 2 A,the user's finger is touching the right area of the circular touchpad208, causing the fast-forward graphical element 202 to becomehighlighted, shaded, or otherwise enhanced. If the user presses down onthe example circular touchpad 208 when one of the correspondinggraphical elements are highlighted corresponding to the circulartouchpad 208, i.e., the fast forward element 202, the back/returnelement 204, the fast reverse element 206, the information element 212,or the play/pause element 214, the universal controlling device 102transmits a second key id to the STB 108, and the STB 108, in response,causes one or more predetermined actions to occur in association withthe second key id, such as to alter the appearance of the selectedgraphical element on the TV 106 and either fast forward TV content, fastreverse TV content, rewind the TV content by 30 seconds, or causeinformation regarding the TV content to be displayed on the TV 106.

Continuing the example from FIG. 2A and assuming the user pressed anarea of the circular touchpad 208 to rewind the TV content, FIG. 2Billustrates how the play graphical element 214 is highlighted when theuser touches (but not presses) the centered key 210. FIG. 2C illustrateshow the STB 108 alters the appearance of the play/pause graphicalelement 214 when the user presses the centered key 210. Also, the STB108 may cause the first consumer electronic template 200 to be removedfrom view on the TV 106 while further stopping the rewind function andproviding the TV content to the user at normal speed.

FIG. 3A is an illustrative example of an embodiment of a consumerelectronic device template, displayed by a smart device when a user iswatching a movie. In this embodiment, the consumer electronic devicetemplate 200, of FIG. 2A, represents a set of control and/or activityelements that are available for a source of the movie, such as a DVDplayer, a ROKU® brand media streamer, etc. The consumer electronicdevice template 200 may be stored in a memory of the source of the movieand/or a “master” smart device, such as the STB 108 and displayed by thesource of the movie or the master smart device when a user touches aninput key on the universal controlling device 102. In anotherembodiment, the templates may be stored on the remote server 120 andaccessed over the wide-area network 118. The term “master” smart devicemay refer to a central consumer electronic device which serves as asource of media content to consumer electronic devices such as TV's,speakers, etc. The master smart device typically comprises wired orwireless input ports to receive entertainment content from a variety ofsources, such as DVD players, media streaming devices such as a ROKU®brand media streamer, STBs, cable TV sources, satellite TV dishes,gaming consoles, etc. A typical master smart device may comprise aYAMAHA® brand RX-A720 AVENTAGE® receiver, a COMCAST® brand STB, etc.

In the embodiment of FIG. 3A, the consumer electronic device template200 for the activity of the user watching a movie comprises an“Information” graphical element 212 associated with a key press of aninput key location “0” on the universal controlling device 102, a “FastForward” graphical element associated with an input key location “3” onthe universal controlling device 102, a “Back” graphical elementassociated with an input key location “5” on the universal controllingdevice 102, a “Rewind” graphical element associated with an input keylocation “7” on the universal controlling device 102, and a “Play/Pause”graphical element associated with an input key location “10” on theuniversal controlling device 102.

FIG. 3B is an illustrative example of another embodiment of a consumerelectronic device template for the activity of watching the TV. In thisembodiment, the consumer electronic device template 302, represents aset of graphical elements that are displayed after a user touches a keyon the universal remote controller 102 when the user activity is“Watching Television”. Similarly, a “Channel up” graphical element isassociated with an input key location “0” on the universal controllingdevice 102, a “Volume up” graphical element is associated with an inputkey location “3” on the universal controlling device 102, a “Channeldown” graphical element is associated with an input key location “5” onthe universal controlling device 102, a “Volume down” graphical elementis associated with an input key location “7” on the universalcontrolling device 102, and a “Channel number graphical element isassociated with an input key location “10” on the universal controllingdevice 102.

FIG. 3C is an illustrative example of yet another embodiment of aconsumer electronic device template. In this embodiment, the consumerelectronic device template 304, represents a set of graphical elementswhen the user activity is “Watching Television”, where the user pressesthe input key location ten (e.g. the Channel number key), and a secondset of graphical elements are determined by the STB 108, to display inthe TV 106. The second set of graphical elements are associated with oneor more input key locations on the universal controlling device 102. Inthis embodiment, input key locations “0” through “10” map to the channelnumbers “0” through “9” and an input key location “10” maps to the“back” function. In some embodiments, the consumer electronic devicetemplate 200 may comprise, two or more layers of associated sets ofgraphical elements.

FIG. 4 is a functional block diagram 400 of one embodiment of theuniversal controlling device 102 as shown in FIG. 1, for controlling oneor more consumer electronic devices. The controlling device 102, in thisexample, comprises a processor 402, an electronic storage medium 404, acommunication interface 406, a user interface 408, at least onetransceiver 410, and at least one transmitter 412.

The processor 402 may be configured to provide general operation of thecontrolling device by executing processor-executable instructions storedin the electronic storage medium 404, for example, executable code. Theprocessor 402 typically comprises a general-purpose microprocessorspecifically adapted for use in the controlling device 102, although anyone of a variety of microprocessors, microcomputers, and/ormicrocontrollers may be used alternatively, chosen based on factors suchas computing power, cost, size, etc.

The electronic storage medium 404 comprises one or more informationstorage devices such as ROM, RAM, Flash, other type of electronic,optical or mechanical electronic storage medium device, or anycombination thereof. The electronic storage medium 404 may be used tostore processor-executable instructions for operation of the controllingdevice 102. It will also be appreciated that some or all of theillustrated electronic storage medium may be physically incorporatedwithin the same IC chip as the processor 402.

As will be understood by those skilled in the art, some or all of theelectronic storage medium 404 may store a plurality of unique key ids,each of the unique key ids associated with a particular input key of theuniversal controlling device 102.

The communication interface 406 comprises one or more data interfacecircuitry, such as well-known Ethernet, Wi-Fi, RF4CE, Bluetooth, or USBcircuitry, that allows wireless communication between the universalcontrolling device 102, and the consumer electronic devices, and in someembodiments, between the universal controlling device 102 and a smartdevice or the wireless router 114 for communication therewith and theserver 120, via the wide-area network 118. In one embodiment, thecommunication interface 406 comprises one or more data interfacecircuitry, such as at least one transceiver 410 and at least onetransmitter 412, that allows communication between coupled smartdevices. In this embodiment, the transceiver 410 may support a firstwireless protocol which is used to communicate with a first smart deviceand the transmitter 412 may support a second wireless protocol which isused to communicate with a second smart device for providing key ids toeach of the smart devices.

The user interface 408 comprises a user input device, for allowing auser to control operations of the universal controlling device 102. Theuser input typically comprises at least one or more input keys ormechanical keys for allowing a user to enter commands or informationinto the universal controlling device 102.

It should be understood that the functional blocks may be coupled to oneanother in a variety of ways other than which is shown in FIG. 4 andthat not all functional blocks necessary for operation of the universalcontrolling device 102 are shown, such as a power supply, a microphone,one or more accelerometers, a multiple-axis gyroscope, a variety ofother transceivers and transmitters each comprising a different wirelessprotocol, for purposes of clarity.

FIG. 5 is a functional block diagram 500 of one embodiment of a smartdevice as shown in FIG. 1, for controlling one or more consumerelectronic devices.

In this embodiment, the smart device comprises a processor 502, anelectronic storage medium 504, a communication interface 506, anoptional user interface 508, and a transmitter/transceiver 510. Itshould be understood that the functional blocks may be coupled to oneanother in a variety of ways other than which is shown in FIG. 5 andthat not all functional blocks necessary for operation of the smartdevice are shown, such as a power supply, a variety of othertransceivers and transmitters each comprising a different wirelessprotocol for purposes of clarity.

The processor 502 is configured to provide general operation of a smartdevice by executing processor-executable instructions stored in theelectronic storage medium 504, for example, executable code. Theprocessor 502 typically comprises a general-purpose microprocessor, suchas an INTEL® CORE™ 17 brand or an AMD K10 brand microprocessor, althoughany one of a variety of microprocessors, microcomputers and ormicrocontrollers may be used alternatively, chosen based on factors suchas computing power, cost, size, etc.

The electronic storage medium 504 comprises one or more informationstorage devices such as ROM, RAM, Flash, other type of electronic,optical or mechanical electronic storage medium device, or anycombination thereof. The electronic storage medium 504 may be used tostore processor-executable instructions for operation of the smartdevice. It will also be appreciated that some or all of the illustratedelectronic storage medium may be physically incorporated within the sameIC chip as the processor 502.

As will be understood by those skilled in the art, some or all of theelectronic storage medium 504 may store instructions or data that isparticular to each type of smart device to be controlled. For example,instructions for the STB 108 may comprise instructions to receivetelevision programming via the communication interface 506, may displayone of the television programs, such as a movie, and display graphicalelements 202, on a display, in accordance with the key ids received fromthe universal controlling device 102. The electronic storage medium 504may additionally store a number of consumer electronic device templates200, each one corresponding to a particular consumer device that may beoperating at a user's home.

The communication interface 506 comprises one or more data interfacecircuitry, such as the transmitter/transceiver 510, Ethernet, Wi-Fi,RF4CE, Bluetooth, or USB circuitry, that allows digital communication,between the smart device and other coupled smart devices, the smartdevice and the universal controlling device 102, via a local areanetwork provided by the wireless router 114, as well as between thesmart device and the server 120, via the wide-area network 118. In thisembodiment, the transmitter/transceiver 510 may support a wirelessprotocol, which is used to receive the key ids from the universalcontrolling device 102, may decode, compress or perform other operationsas necessary for displaying the graphical element 202, in a displaydevice.

The user interface 508 comprises a user input device and/or a useroutput device, for allowing a user to control operation of the smartdevice. The user input typically comprises one or more pushbuttons,keys, a touchscreen display, etc. for allowing a user to enter commandsor information into the smart device. The user output device typicallycomprises a display screen, a touchscreen display, lights, amplifiedsound bars, etc. for presenting media content to a user asdesired/needed.

It should be understood that the functional blocks may be coupled to oneanother in a variety of ways other than which is shown in FIG. 5 andthat not all functional blocks necessary for operation of a smart deviceare shown, such as a power supply, a variety of other transceivers andtransmitters each comprising a different wireless protocol, for purposesof clarity.

FIG. 6 is a flow diagram for a method for controlling one or moreconsumer electronic devices, performed by a smart device in cooperationwith a universal controlling device. The method is implemented by theprocessor 502 located within a smart device, such as the STB 108, byexecuting processor-executable instructions stored in the electronicstorage medium 504. It should be understood that in some examples, notall of the steps shown in FIG. 7 are performed and that the order inwhich the steps are carried out may be different. It should be furtherunderstood that some minor method steps, which would be known to thoseof ordinary skill in the art, have been omitted for purposes of brevity.

At a block 600, a user of the universal controlling device 102 touchesan input key on the universal controlling device 102, as part of theuser interface 408. The input key comprises a switch that is activatedwhen pressed, as well as a touch sensor to determine when the key istouched. In response, the touched input key provides a signal to theprocessor 402 indicative of a particular key location that was touched.In another embodiment, the processor 402 monitors some or all of theinput key locations and associated touch sensors to determine when oneof the input keys exhibits a changed characteristic as a result of theuser touching one of the input key locations, such as the capacitance ofa touch sensor.

At a block 602, the in response to the user touching one of the inputkey locations, the processor 402 determines which touch sensor exhibiteda change in characteristic. In one embodiment, the processor 402,determines which input key location was touched by determining a changein capacitance associated with one or more of the touch sensors.

At a block 604, the processor 402 receives a first and a second datafrom a first and a second touch sensor.

At a block 606, the processor 402 determines if one or more touch sensorsignals were received, via the user interface 408. In one embodiment,the processor 402 receives a first data from a first touch sensor andmay receive a second data from a second touch sensor where the seconddata may comprise a higher increase of the parasitic capacitance toground value than the parasitic capacitance to ground value of the firstdata. In this embodiment, the second data is utilized to determine, bythe processor 402, that the higher increase of the parasitic capacitanceto ground represents that the user is touching the second input keylocation and not the first input key location. In one embodiment, theprocessor 402 receives an altered value of the mutual coupling betweenrow and column electrodes to determine the input key location that theuser is touching.

In one embodiment, at a block 608, the processor 402 determines a firstorientation data of the universal controlling device 102, where thefirst orientation data corresponds to a first input key location on theuniversal controlling device 102 as described in FIG. 1. In one example,the processor 402 may receive orientation data from one or moreaccelerometers and/or one or more multiple-axis gyroscope where thefirst orientation data corresponds to a first input key location. It isto be understood that while orientation sensors are described as anaccelerometer and a multiple-axis gyroscope orientation sensors mayinclude, but need not be limited to, any sensor type that is capable ofdetecting a change in alignment of the universal controlling device 102,depending on the direction that the universal controlling device 102, isheld, and/or, generally, any sensor that is also capable ofcommunicating information with controlling type devices and/or othersmart devices.

At a block 610, the processor 402, determines a first key id associatedwith a first input key that was touched by retrieving a first key idfrom the memory 404. For example, a user of the universal controllingdevice 102 may touch the first input key location on the universalcontrolling device 102 (as opposed to pressing the input key), which ata block 612, causes the processor 402 to transmit the input key id and,in one embodiment, orientation data via the transmitter/transceiver 410and/or 412, to the smart device, in this case the STB 108.

At a block 614, the processor 502 of the STB 108 receives the first keyid via the transmitter/transceiver 510 from the universal controllingdevice 102. In one embodiment, the processor 502 may also receiveorientation data, via the transmitter/transceiver 510, from theuniversal controlling device 102. In this embodiment, the universalcontrolling device 102, provisions the processor 502, with orientationdata and causes the processor 502, to determine a first orientation ofthe universal controlling device 102. In response to the processor 502,determining a first orientation, the processor 502, sends key idreprogramming instructions, via the communication interface 508, to theuniversal controlling device 102, where the universal controlling device102, in real time, reprograms the unique key ids for each input keylocation to align with a first orientation of the universal controllingdevice 102. For example, the universal controlling device 102, mayinclude more than one sensors that detect that the user is holding theuniversal controlling device 102, with the user's right hand. When theuser picks up the universal controlling device 102, the universalcontrolling device 102 sends the orientation data to the processor 502,where the processor 502, then determines that the input key locatedfurthest from the user's hand in the Y-axis is the top most input keyand sends, via use of the communication interface 508, reprogramminginstructions to the universal controlling device 102. Similarly, allinput keys on the universal controlling device 102, may be reprogrammedbased on the orientation of the universal controlling device 102.

At a block 616, the processor 502, determines a first consumerelectronic device template 200 from a plurality of consumer electronictemplates stored in the electronic storage medium 504, the firstconsumer electronic device template comprising a set of graphicalelements 202, where each graphical element 202 is associated with anaction of a particular consumer electronic device or smart device. Inanother embodiment, the plurality of consumer electronic templates arestored remotely, for example at the remote server 120.

At a block 618, the processor 502, transmits, via the communicationinterface 506, a set of the graphical elements 202 to a display devicesuch as the TV 106, in some embodiments, via use of the HDMI cable 112connected to the STB 108. In another embodiment, the graphical elements202 are displayed on a display device associated with the smart devicethat is in communication with the universal controlling device 102. Forexample, the TV 106 may be in direct communication with the universalcontrolling device 102. the processor 502 within the TV 106 may displaythe graphical elements 202 (i.e., the selected electronic consumerdevice template) on the display of the TV 106.

At a block 620, the processor 502 receives a second key id via thetransmitter/transceiver 510 from the universal controlling device 102,representative of an input key on the universal controlling device 102that was pressed by the user. For example, an input key on the universalcontrolling device 102 may be pressed, via use of a dome structurebeneath the capacitive key structure, for sending a second key idassociated with the pressed input key to the STB 108. In someembodiments, the second key id may be different than the first key id.

At a block 622, the processor 502 determines an action associated withthe second input key, based on the particular consumer electronic devicetemplate presently displayed to the user in the first graphical element202. For example, when a TV template is being displayed and the secondkey id corresponds to a user pressing a particular area of the circulartouchpad 208, the processor 502 may determine that a fast-forwardfunction is desired by the user. When a different consumer devicetemplate is displayed and the same second key id is received, adifferent function may be invoked, such as to pause content currentlybeing displayed by a ROKU® brand media streamer, for example. In oneembodiment, the second key id may comprise a key down event, a key pressevent or any registered key value associated with the input key that waspressed by the user.

At a block 624, the processor 502, sends tone or more operationalcommands associated with one of the actions, where the operationalcommands are in conformity with a communication protocol used by aparticular consumer electronic device associated with the template, tothe active electronic consumer device for executing the operationalcommands. In some embodiments, an operational command may comprise amacro operation (e.g. two or more operational commands that may beexecuted by one or more consumer electronic devices).

In some embodiments, a real-time status/content detection operation isperformed by a smart device comprising a Universal Control Engine, andthe detected available media content is used to determine which consumerelectronic device template to display. In these embodiments, thefunctionality for gathering the identification of available mediacontent from detected smart devices or detected consumer electronicdevices located at an end-user's home may be performed as described inU.S. patent application Ser. No. 15/647,947 filed on Jul. 12, 2017 andwhich is incorporated herein by reference in its entirety.

In another embodiment, the STB 108, may perform a detection operation,such as a detect/request the active source/sink/port on theaudio-video/communications bus status request, to detect an active HDMIinput of an active consumer electronic device. The STB 108 thendetermines a consumer electronic device connected to the active HDMIinput and, in turn, determines the consumer electronic device template200 associated with the active consumer electronic device.

For example, when the universal controlling device 102 is picked up,tapped or moved by a user and the user's finger touches an input key onthe universal controlling device 102, the universal controlling device102 sends a key id to the STB 108, where in response to receiving thekey id, the STB 108 determines a particular one of the consumerelectronic device templates 200 that is associated with a consumerelectronic device that is presently determined to be active by thedetection operation discussed above. The STB 108 then selects theconsumer electronic device template 200 based on the active consumerelectronic device and sends the elected consumer electronic devicetemplate to the TV 106, via a wired or wireless communication protocol.The TV 106, in response to receiving the selected consumer electronicdevice template, displays the template, indicative of the availablecontrols and/or activities available to the user for selection. Suchactivities may comprise for example, “Play”, “Pause”, “Fast-forward”,“Back” and “Rewind” when the user is watching a movie or when the useris listening to a music streaming device. In this example, the selectedconsumer electronic device template comprises graphical elements relatedto the TV 106 and may be as highlighted or shaded when the user hastouched an associated input key. In this embodiment, a key touch is usedfor navigation of the set of graphical elements.

In one embodiment, the consumer electronic device templates 200, may bestored locally on the STB 108, or may be automatically provisioned tothe STB 108, by the coupled server 120 having access to the database122, via use of the wide area network 118.

In one embodiment, the detection operation performed by the STB 108comprises sending signals to one or more smart devices and/or consumerelectronic devices discovered during the discovery process describedearlier herein. Such a discovery process causes the detected devices toprovide their device information, such as EDID, CEC, vendor name, devicetype, device status, installed apps, current media content being playedon the device, media content logos, Infoframe, SSDP, MDNC, IP mDNSService List, supported wireless protocols such as VoIP, IP, Smart Wave(S-WAVE), Wi-Fi, Bluetooth low energy (BLE), RF4CE, ZigBee, Z wave,Infra-red, etc., to a requesting smart device. For example, thedetection operation or provisioning process may be used to determinesmart devices and/or consumer electronic devices comprising the sameoperational capabilities. Further details of such device detection maybe found in U.S. Pat. Nos. 8,812,629, 8,558,676, 8,659,400, 8,830,074,8,896,413, 9,437,105, 9,449,500 and 9,019,435 all of which areincorporated herein by reference in their entirety.

In any of the examples, the functionality for gathering theidentification of smart devices information from smart devices locatedat an end-user's home may be performed by the Universal Control Engine(UCE) 100 as described in U.S. Pat. No. 9,215,394 which patent isincorporated herein by reference in its entirety. In one example, when asmart device, comprising the UCE 100 is initially powered on, anautomatic detection operation or provisioning process may be initiatedfor identifying or detecting consumer electronic devices and/or smartdevices on the same local network as the smart device containing the UCE100. Alternatively, the detection operation may be initiated by an inputkey press on the universal controlling device 102. Such a provisioningprocess is described in U.S. Pat. No. 9,307,178 which is alsoincorporated herein by reference in its entirety.

It is to be understood that the universal controlling device 102, asshown, FIG. 1 may include a plurality of other functions such as motionsensor-enabled capabilities, hand-gesture recognition and may includethe capability to display images, such as logos, alphanumeric text, etc.Such controlling devices may cooperate with one or more apps, to controla smart device. Furthermore, the universal controlling device 102, maycooperate with one or more smart devices each comprising a computingclient, such as in a client-server-model to provision and/or to controla smart device.

In one embodiment, the universal controlling device 102, having anassociated control app and coupled to the server 120, sends a request toa smart device, comprising the associated app, to perform a localoperation to dynamically scan the local-area network for connected smartdevices when an input key on the universal controlling device 102, istouched or pressed by the user, where the app operates as anintermediary device for communications between the universal controllingdevice 102, and the smart device. In this embodiment, the app mayreceive instructions from the smart device to reprogram the unique keyids on the universal controlling device 102, of each of the input keysto align with the first orientation of the universal controlling device102.

In another embodiment, the universal controlling device 102, comprisesmechanical keys for allowing a user to enter commands or informationinto the universal controlling device 102. In this embodiment, themechanical keys, are without key labels and may be arranged in a row andcolumn key layout, such as in a four key by four key matrix keypad.

While illustrated as the TV 106, the STB 108, and the streamer 110, itis to be understood that smart devices may include, but need not belimited to, various televisions, VCRs, DVRs, DVD players, cable orsatellite converter set-top boxes, amplifiers, CD players, gameconsoles, home lighting, smart wireless hubs, drapery, fans, HVACsystems, personal computers, wearable health monitoring devices, or,generally, any consumer product capable of communicating withcontrolling type devices and/or other smart devices. In otherembodiments, smart devices are not internet capable, but are capable ofcommunicating with consumer electronic devices via wired or wirelessmeans.

While described as an active HDMI input, it is to be understood that anactive input includes any active source/sink/port on theaudio-video/communications bus status either wired or wirelesslyconnected to the smart device which initiated the status request.

It is also to be understood that the universal controlling device 102,may be partially configured at a factory with one or more key idspreinstalled. When initially powered on, the universal controllingdevice 102, may be configured to automatically communicate with apredetermined smart device, such as the STB 108, such as when theuniversal controlling device 102 and the STB 108 are paired with eachother out of the-box. Similarly, an automatic pairing operation may beperformed when the end user presses a first key, on the universalcontrolling device 102, for initiating a communication with the STB 108.

While various concepts have been described in detail, it will beappreciated by those skilled in the art that various modifications andalternatives to those concepts could be developed in light of theoverall teachings of the disclosure. Further, while described in thecontext of functional modules and illustrated using block diagramformat, it is to be understood that, unless otherwise stated to thecontrary, one or more of the described functions and/or features may beintegrated in a single physical device and/or a software module, or oneor more functions and/or features may be implemented in separatephysical devices or software modules. It will also be appreciated that adetailed discussion of the actual implementation of each module is notnecessary for an enabling understanding of the invention. Rather, theactual implementation of such modules would be well within the routineskill of an engineer, given the disclosure herein of the attributes,functionality, and inter-relationship of the various functional modulesin the system. Therefore, a person skilled in the art, applying ordinaryskill, will be able to practice the invention set forth in the claimswithout undue experimentation. It will be additionally appreciated thatthe particular concepts disclosed are meant to be illustrative only andnot limiting as to the scope of the invention which is to be given thefull breadth of the appended claims and any equivalents thereof.

All patents cited within this document are hereby incorporated byreference in their entirety.

What is claimed is:
 1. A non-transitory computer readable media havingstored thereon instructions executable by a processing device of a firstappliance, wherein the instructions, when executed by the processingdevice of the first appliance, cause the first appliance to performsteps comprising: receiving from each of a plurality of media sourceappliances connected to a respective one of a plurality of input portsof the first appliance data indicative of at least one available mediacontent; using the data indicative of the available at least one mediacontent received from each of the plurality of media source appliancesto cause a plurality of display elements to be displayed in a displayassociated with the first appliance; receiving a data that isrepresentative of a one of the plurality of display elements; using thereceived data that is representative of the one of the plurality ofdisplay elements and information indicative of the plurality of displayelements caused to be displayed in the display associated with the firstappliance to determine a command to be transmitted from the firstappliance to a target appliance; and transmitting to the targetappliance, using a communication protocol recognizable by the targetappliance, the determined command for the purpose of controlling acorresponding operation of the target appliance.
 2. The non-transitorycomputer readable media as recited in claim 1, wherein the targetappliance is a one of the plurality of media source appliances.
 3. Thenon-transitory computer readable media as recited in claim 2, whereinthe command comprises a command to cause the target appliance to performa media playing operation.
 4. The non-transitory computer readable mediaas recited in claim 1, wherein the data that is representative of a oneof the plurality of display elements is received from a controllingdevice in communication with the first appliance.
 5. The non-transitorycomputer readable media as recited in claim 4, wherein the controllingdevice uses a radio frequency communications protocol to communicatewith the first appliance.
 6. The non-transitory computer readable mediaas recited in claim 4, wherein the controlling device uses an infraredcommunications protocol to communicate with the first appliance.
 7. Thenon-transitory computer readable media as recited in claim 4, whereinthe data that is representative of a one of the plurality of displayelements comprises a key id that is representative of an activated inputelement of the controlling device.
 8. The non-transitory computerreadable media as recited in claim 1, wherein the data that isrepresentative of a one of the plurality of display elements comprisesdata indicative of a spoken utterance that is representative of the oneof the plurality of display elements.
 9. The non-transitory computerreadable media as recited in claim 8, wherein the data that isrepresentative of a one of the plurality of display elements is receivedfrom a controlling device in communication with the first appliance. 10.The non-transitory computer readable media as recited in claim 1,wherein the communication protocol recognizable by the target appliancecomprises a wireless communication protocol.
 11. The non-transitorycomputer readable media as recited in claim 1, wherein the communicationprotocol recognizable by the target appliance comprises a wiredcommunication protocol.
 12. The non-transitory computer readable mediaas recited in claim 9, wherein the command is transmitted to the targetappliance via use of a one of the plurality of input ports of the firstappliance.